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Abstract 

Background: Research on the neurophysiological correlates of visuomotor integration and learning (VMIL) has largely 
focused on identifying learning-induced activity changes in cortical areas during motor execution. While such studies 
have generated valuable insights into the neural basis of VMIL, little is known about the processes that represent the 
current state of VMIL independently of motor execution. Here, we present empirical evidence that a subject's 
performance in a 3D reaching task can be predicted on a trial-to-trial basis from pre-trial electroencephalographic 
(EEG) data. This evidence provides novel insights into the brain states that support successful VMIL 

Methods: Six healthy subjects, attached to a seven degrees-of-freedom (DoF) robot with their right arm, practiced 
3D reaching movements in a virtual space, while an EEG recorded their brain's electromagnetic field. A random forest 
ensemble classifier was used to predict the next trial's performance, as measured by the time needed to reach the 
goal, from pre-trial data using a leave-one-subject-out cross-validation procedure. 

Results: The learned models successfully generalized to novel subjects. An analysis of the brain regions, on which the 
models based their predictions, revealed areas matching prevalent motor learning models. In these brain areas, the 
a /ii frequency band (8-14 Hz) was found to be most relevant for performance prediction. 

Conclusions: VMIL induces changes in cortical processes that extend beyond motor execution, indicating a more 
complex role of these processes than previously assumed. Our results further suggest that the capability of subjects to 
modulate their a/fi bandpower in brain regions associated with motor learning may be related to performance in 
VMIL. Accordingly, training subjects in a//x-modulation, e.g., by means of a brain-computer interface (BCI), may have a 
beneficial impact on VMIL. 

Keywords: Visuomotor integration and learning, Motor learning, Brain-computer interface, BGI, EEG, Performance 
prediction 



Background be due to the fact that early learning is closely related 
Motor learning of novel kinematic and/or dynamic envi- to attention and relies on explicit working memory and 
ronments can be categorized by learning phase and learn- forming new associations between visual cues and motor 
ing form [1]. The temporal course of motor learning commands [7]. Other areas involved in early stages 
is oftentimes divided into three phases, an early- (slow of motor learning include (pre-)motor areas [5,7] and 
performance, close sensory guidance), an intermediate- superior-posterior parietal cortex [8]. In later stages, pre- 
(gradual learning, increase in speed) and an advanced frontal activation shifts more to the left hemisphere 
phase (skillful and automatized movements) [2]. In the [6,7]. This left-hemispheric dominance appears to be 
early stage of motor learning prefrontal areas play a independent of the side used for training [9]. Further- 
key role, especially the dorsolateral frontal cortex and more, the learning process can be categorized into two 
the right prefrontal cortex [3-6]. This key role might forms: explicit learning, in which subjects consciously 
try to learn a task relying on previous experiences, and 

^Correspondence: tmeyer@tuebingen.mpg.de implicit learning, which takes place unintentionally and 

1 Department Empirical Inference, Max Planck Institute for Intelligent Systems, . , 

Tubingen, Germany ' unconsciously. 
Full list of author information is available at the end of the article 



© 2014 Meyer et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative 
BiolVlGCl C6ntf3l Commons Attribution License (http://creativecommons.0rg/licenses/by/2.O), which permits unrestricted use, distribution, and 
reproduction in any medium, provided the original work is properly credited. 



Meyer et al. Journal ofNeuroEngineering and Rehabilitation 201 4, 1 1 :24 
http://www.jneuroengrehab.eom/content/1 1/1/24 



Page 2 of 13 




Figure 1 Subject wearing an EEG-cap while being attached to 
the seven DoF Barrett WAM arm (subject consented to this 
image being used in this publication). 



To date, one widely accepted model of motor learn- 
ing is the one proposed by Hikosaka and colleagues 
[10]. This model comprises two parallel loop cir- 
cuits, one responsible for learning spatial features 
(frontoparietal-associative striatum-cerebellar circuit), 
and the other one responsible for learning motor features 
(motor cortex-sensorimotor striatum-cerebellar circuit). 
Transformations between the two loops take place in the 
supplementary motor area (SMA), the pre-SMA and the 
premotor cortices. 

In this article, we investigate whether the involvement 
of these areas in motor learning is restricted to peri- 
ods of actual motor execution, or if they also represent 
the current state of motor learning when subjects are 
either at rest or are preparing for an upcoming movement. 
We present results of an EEG study on explicit learning 
of a sensorimotor task, and provide empirical evidence 
that cortical structures known to be involved in motor 
learning do indeed provide information about the actual 
progress of motor learning, i.e. they predict the precision 
of an upcoming movement. We discuss the implications of 
these findings for motor learning in general and the use of 
brain-computer interfaces (BCIs) for motor rehabilitation 
in particular. 

Methods 

Subjects 

Six healthy subjects (3 male, 3 female; mean age 29.5=b4.5), 
recruited from the local student body, participated in the 
present study. All subjects were right-handed and thus 
conducted the study with their right arm. All subjects 
except subject three were naive to the task. Subject three 
had participated in a similar experiment with two dimen- 
sional reaching movements. All subjects gave informed 
consent in accordance with guidelines set by the Max 
Planck Society. The paradigm and experimental setup of 
this study have been approved by an ethics committee of 
the Max Planck Society. 

Study design 

The subjects right arm was attached to a seven degrees of 
freedom (DoF) robotic arm (Figure 1) facing a feedback 
screen at a distance of approximately 1.5 meters. Due to 
the robotic arms DoF, the subjects were able to perform 
a large variety of natural movements. The robot com- 
pensated gravity for its own weight, therefore the subject 
required only negligible forces to move his and the robots 
arm. In this study the robotic arm was used in a purely 
passive way, whereas the system is designed in a way that 
the robot can disturb, influence or support movements 
(see [11] for a more detailed description of the robotic 
setup). 

The goal of each trial was to move the robot arm to 
reach a target visualized in 3D on a computer screen and 



thereafter back to the initial starting position. For each 
trial, the target was chosen from a set of pre-defined 
targets. The screen continuously displayed the current 
position of the robotic arms end-effector and the target 
position. Each subject performed 200 trials divided into 
four blocks of 50 trials, interleaved with a brief one minute 
intermission. EEG data was continuously recorded during 
the experiment. 

Trial design 

Table 1 provides an overview over the four phases that 
each trial consisted of. An overview over the visualization 
is provided in Figure 2. Each trial started with the baseline 
phase, for which the subject was instructed to do noth- 
ing and no feedback was shown (c.f. Figure 2(a)). After 
5 seconds, the phase switched to the planning phase for 
which the subject was instructed to plan the movement, 
but not yet move. This phase change was indicated by dis- 
playing the current end-effectors position as blue ball and 
the target as yellow ball (c.f. Figure 2(b)). The phase lasted 
2.5-4 seconds, with the duration chosen randomly from 
an uniform distribution. The experiment then continued 
with the go phase, which was indicated by switching the 
targets color from yellow to green (c.f. Figure 2(c)). The 
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Table 1 Experiment phases 

Phase Duration Instruction Visualization 

Baseline 5 sees Do nothing Neither target nor current position was shown 

Planning 2.5-4 sees Plan path Current position was shown as blue ball, target 

position as yellow 

Go Max 10 sees Reach target Current position was shown as blue ball, target 

position as green 

Return to start Max 10 sees Return to start current position was shown as blue ball, initial 

position as green 

This table describes the phases of the experiment regarding their name, duration, the instruction provided to the subject and the visual feedback provided. 



subject was instructed to bring the current hand posi- 
tion in congruence with the target position, i.e. reach for 
the target. A reaching movement was considered com- 
plete when the subject moved the end-effector within 1.5 
cm of the target location, or if the subject exceeded a 
ten seconds time limit. In either case, the green ball at 
the target position disappeared and was replaced by a 
green ball at the initial starting position of the end-effector 
(c.f. Figure 2(d)). This event started the last phase - return 
to start - for which the subject was instructed to return 
to the starting position. When the subject moved the end- 
effector to within 4 cm of the initial position or a time 
limit of ten seconds ran out, the robot arm gently pulled 
the end-effector to its precise starting position for the next 
trial. 

In each trial, a different target location was chosen 
from a sphere located in front of the subject. In order to 
determine a range of reachable targets, while considering 
the subjects individual physical differences, each subject 
determined the center and radius of the sphere prior to 
initiation of the first trial by moving their arm to multiple 



comfortable positions in front of their body. The sphere 
was defined as the minimum sphere that enclosed 90% of 
all the positions visited while the subject moved between 
these comfortable positions. In this study, the radii varied 
from 5-9 cm. 

Visualization 

The visual feedback consisted of a virtual box in which 
the current end-effectors position and the target posi- 
tion were displayed as colored balls (c.f. Figure 2). The 
bar at the top of the screen provided information about 
the depth of the balls. Both balls were plotted on this 
bar according to their z coordinate. A position far- 
ther on the left on the bar indicated a position closer 
to the subject. This bar was added to enhance depth- 
perception when the target and current position over- 
lapped on the screen (c.f. Figure 2(e)). This problem 
originates in projecting a three dimensional task onto a 
two dimensional fixed plane. For the same reason poles 
were added to the balls. These poles provided informa- 
tion about the projected location of the balls on the 




(a) (b) (c) 




(d) (e) 

Figure 2 Visual feedback, (a) Feedback shown in the baseline phase, (b) Feedback shown in the planning phase, (c) Feedback shown in the go 
phase, (d) Feedback shown in the return to start phase, (e) The figure shows a state in which the depth bar provides information which is otherwise 
not easy to see. 
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ground plane, thus making it easier to estimate depth and 
distances. 

Data acquisition 

Throughout the study, a 120-channel EEG was recorded 
at 1 kHz sampling rate, using active EEG electrodes 
and a QuickAmp amplifier (BrainProducts, Gilching, 
Germany). Electrodes were placed according to the 
extended 10-20 system, with Cz as the initial reference 
electrode. All data were re-referenced to common average 
reference offline. 

To track each subjects learning process over the course 
of the experiment, the normalized time-to- target (TTT) 
for each trial was computed, i.e. the time required from 
the instruction to initiate the movement to reaching the 
target, divided by the distance from starting position to 
target position. 

Data analysis 

In this section, we describe our data analysis, which 
was conducted to examine whether TTT can be pre- 
dicted from EEG signals originating in the baseline or 
planning phase. We employed a random forest model 



and a leave-one-subject-out cross-validation for this 
purpose. 

Time to target prediction 

In the following, we investigate whether TTT can be pre- 
dicted on a trial-to-trial basis from EEG recorded in the 
upcoming targets planning or baseline phase. To do so, 
we separated the data into group-wise (ideally) statisti- 
cally independent components (ICs). This was done by 
first high-pass filtering each subjects raw data at 3 Hz 
using a third-order Butterworth filter. The data of all sub- 
jects were pooled and reduced to 64 principal components 
before applying a second-order blind identification algo- 
rithm (SOBI) [12]. We inspected each IC manually and 
rejected those which were not of cortical origin or did 
not contain EEG-like spectral densities. The topographies 
of the remaining ICs are shown in Figure 3. We com- 
puted log-bandpower of each non-artifactual IC in each 
trial in five frequency bands (based on the raw data using 
a FFT in conjunction with a Hann window): 8 (0.1-4 Hz), 
0 (4-7 Hz), a/fi (8-14 Hz), p (20-30 Hz), and y (55-85 
Hz). We low-pass filtered these bandpowers in the trial 
domain with .1 radians since we were mostly interested in 




Figure 3 All non-rejected ICs. The figure shows all ICs that were kept after rejecting non cortical ICs. 
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slow changes in contrast to fast variations (see Results and 
discussion section). 

These bandpowers served as input variables to a ran- 
dom forest trained on five subjects, to predict the normal- 
ized TTT on the remaining subject. With 14 ICs and five 
different frequency bands, this random forest model oper- 
ates on a 70 dimensional feature space. Cross-validation 



within a subject requires data to be independent and iden- 
tically distributed (IID) [13], which is not the case for 
movement performance data. We therefore employed a 
leave-one-subject-out cross-validation instead. This was 
done in turn for all subjects. 

Since the input variables are low-pass filtered, our 
model cannot capture high-frequency changes. Thus, 
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Figure 5 PSD of Mean TTT. This figure shows the PSD of the mean TTT. 
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quality estimation of the prediction using standard R 2 , 
the coefficient of determination, would be underrated. To 
provide a better quality estimate, the following modifica- 
tion of R 2 was used. Let R 2 (x, y) denote the usual defini- 
tion of R 2 with the observed values x and the modeled 
values y, and let LP(x, c) denote variable x low-pass fil- 
tered with a cut-off frequency of c radians. R 2 (x, LP(x, c)) 
reflects the amount of variance that can be explained by 
using only the low-frequency components of x and thus 
represents the maximum achievable value for a random 
forest model based on frequencies less or equal than c 
radians. We then define 



R mod^y) 



R 2 (x,y) 



R 2 (x,LP(x,c))' 



(1) 



Thus R 2 mod ranges from — oo to 1, where a value less 
than 0 means that additional variance was introduced, 
e.g., by using a completely random prediction. All val- 
ues over 0 are desirable, with 1 being the best achievable 
value, i.e. explaining all variance that can be explained by 
low frequencies. The specific value of c used is explained 
in chapter Results and discussion. We calculated R 2 mod 
between predicted and actual TTT, and tested group level 
significance with a permutation test. For this test, the trial 
order was permuted independently for each subject, and 
the average of R 2 mod over all subjects was calculated. This 
was repeated 10,000 times and significance was estimated 



by using the relative position of the real average R^ mod in 
comparison to the permutation based R 2 mod values. 

Model interpretability 

Although random forests are based upon decision trees, 
random forests lack the interpretability of these. As a con- 
sequence, Breiman devised a measure that reflects the 
importance of a variable for an accurate prediction [14]. 
Computing variable importance is based on measuring 
the accuracy drop in case the values of one variable are 
permuted. 

After using variable importance to analyze which input 
variables have a large influence on the prediction, one can 
use the learned model to see how these variables affect the 
predictions, as described in the following approach. In the 
context of this study, a variable v refers to the bandpower 
of one IC in one specific frequency band. A reasonable 
range of values to analyze is defined by the minimum and 
maximum value that was encountered in the training data 
for the analyzed variable v. Any value outside this range 
has the same effect on the prediction as the minimum 
or maximum value. To determine the average effect of a 
variable v on the prediction the following steps are per- 
formed: (1) in all training data, replace the measured value 
of variable v by v, (2) use the model to predict the output 
for the modified data, (3) the average output provides an 
estimate of the effect of setting v to v on the prediction. 
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Figure 7 Normalized TTT of each subject. This figure shows each subjects' ^Tlowpass filtered at .1 radians and normalized. 



Once a sufficiently sized subset of the range is analyzed, 
this provides an assessment of the variables influence on 
the prediction. 

In the context of this study, this approach was used to 
estimate how frequency-specific bandpower changes in 
cortical areas influence the prediction of TTT. 



Results and discussion 

The average time a subject needed to complete the exper- 
iment was 60.17 minutes with a standard deviation of 1.18 
minutes. We observed a continuous decline in TTT over 
the course of the experiment, reflecting successful VMIL 
processes (Figure 4). This trend is captured by the low 
frequencies of the TTTs' power spectral density (PSD) 
(Figure 5). After analyzing the power spectral densities of 
each subjects TTT (Figure 6), we concluded that the most 
important frequency components exist below .1 radians. 
This value was then used as value c for low-pass filter- 
ing the TTT, as described in the data analysis section. 
Figure 7 shows the TTT of each subject low-pass filtered 
at .1 radians. We want to point out that due to low-pass 
filtering the term predicting could be misleading. For the 
purpose of text flow and as this term is oftentimes just 
referring to a models output, we will stick to the term 
prediction. 



Time to target prediction 

Table 2 shows R 2 mod for all subjects with a model trained 
on data from the baseline or the planning phase, as pre- 
viously described. Significance values on a group level are 
provided, which were computed as described in the previ- 
ous section. Subjects three and five show low R 2 mod values 
for both phases. Subject five told us after the experiment 
that he was tired during the experiment and showed signs 
of fatigue. This information is in congruence with his TTT 
(c.f. Figure 7). Since his prediction model was based on the 
other subjects TTT and brain signals - and they did not 
show signs of fatigue - this might be the reason for his low 



Table 2 Prediction quality estimation 


Subject 


Baseline 


Planning 


1 


0.03 


0.19 


2 


0.23 


0.45 


3 


-0.13 


-0.53 


4 


0.10 


0.16 


5 


-0.10 


-0.09 


6 


0.04 


0.18 



Group median 



0.03 (p-value< 0.001) 



0.17(p-value< 0.001) 



This table provides the R 2 mod value for models trained with leave-one-subject-out 
cross-validation on data from the respective phase. 
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Subject 3 




200 



Figure 8 Prediction subject 3 planning. The figure shows the predicted values (dashed line) and the lowpass filtered real values (solid line) for 
subject 3 (planning phase, R 2 mod is -0.53). 



R 2 mod value. Subject three had previous experience with 
this kind of experiment and showed only little signs of 
improvement after a phase of familiarization. This existing 
experience might explain his low R 2 mod value. The group 
level permutation test rejected the null-hypothesis that 
the temporal structure of the features does not provide 



any information on the current state of VMIL for data 
from the baseline phase (p < 0.001) as well as from the 
planning phase (p < 0.001). 

Figures 8, 9 and 10 provide examples of a low (-0.53), 
near zero (0.03) and high (0.45) R 2 mod value. Fi gure 8 shows 
that already a small amount of values contradicting the 
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Figure 12 Importance planning. The figure shows the variable importance values for all ICs in the planning phase per subject. 



general trend leads to a low R 2 mod value. Figure 9 shows a 
prediction oscillating around the mean value, correspond- 
ing to a $ mod value near zero. Figure 10 shows an example 
for a moderately high, positive R 2 



Source localization and relation to existing motor learning 
models 

In order to identify cortical areas relevant for TTT- 
prediction, the variable importance values of the random 
forests were investigated for each subjects model (Figures 
11 and 12). 

For the baseline phase, the /x band of IC 11 shows the 
highest importance values, consistent across subjects. The 
corresponding ICs' topographies (cf. Figure 4) were pro- 
jected back to the cortical level using the BrainStorm 



toolbox [15]. For this purpose a distributed source model 
with minimum-norm estimation was selected, based on 
standard electrode locations and a standard head model. 
Figure 13 shows the source localization result for IC 11. 
This IC contains strong weights in prefrontal areas, pos- 
sibly focused in the dorsolateral prefrontal cortex. It is 
commonly reported that the dorsolateral prefrontal cor- 
tex is involved in the initial stages of explicit motor 
learning, due to its role in sensorimotor association and 
working memory [4,6]. Figure 14 shows the source local- 
ization results for ICs 1, 3 and 9, which are the most 
relevant ICs in the planning phase. These ICs are pri- 
marily localized to parietal cortex, but also exibit activity 
in preSMA, SMA, primary somatosensory cortex, asso- 
ciative visual cortex (V3, V4, V5), prestriate cortex (V2) 
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IC 1 



IC 3 



IC 9 




Figure 14 Source localization ICs 1, 3, 9. The figure shows the source localizations of ICs 1, 3, 9 (left view, right view, top view). 



and somatosensory association cortex. Since the plan- 
ning phase provided information about the next target, 
the contribution of these regions to the prediction is in 
congruence with Hikosakas model, stating that preSMA 
and SMA are responsible for spatiomotor conversion 
processes. The activity in the posterior parietal cortex 



confirms the finding that this region is involved in motor 
planning [8]. 

Due to pooling the EEG data before separating ICs one 
could argue that a small subset of subjects could domi- 
nate certain ICs and therefore distort group effects. This 
argument is weakened by the fact that the most important 
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ICs are consistent across subjects, as seen in Figures 11 
and 12. 

Feature influence on prediction 

For both the baseline- and the planning phase, the ot/ii 
frequency band was found to be most relevant. To ana- 
lyze the relation between this band and TTT prediction, 
we examined the effect of changing the bandpower in 
the aforementioned ICs as described in section Model 
interpretability \ The results are shown in Figure 15. They 
indicate that enhanced bandpower in this frequency band 
in the previously mentioned regions is related to better 
movement performance. 

Conclusions 

Previous research on VMIL concentrated on investigat- 
ing learning-induced changes in brain activity during 
motor execution. In our study we presented empirical 
results indicating that motor performance can be pre- 
dicted from pre-trial EEG signals, thus identifying brain 
regions not only actively involved in motor learning, 
but furthermore providing information about the current 
learning progress. Areas found to be involved include the 
dorsolateral prefrontal cortex, preSMA, SMA, primary 
somatosensory cortex, V2, V3, V4, V5 and somatosensory 
association cortex. These results might provide starting 
points for enhancing motor learning and increasing motor 
rehabilitation performance, e.g., by neurofeedback [16] or 
with a direct stimulation as is the case in transcranial 
direct current stimulation [17]. 

Implications for motor imagery based BCI stroke 
rehabilitation 

While initially conceived as communication devices, 
brain-computer interfaces (BCIs) have recently attracted 
attention as potential tools for stroke rehabilitation 
[16,18-20]. Here, the central idea is to train patients in 
modulating sensorimotor-rhythms (SMRs) by real-time 
neurofeedback, as the extent of SMR-modulation has 
been found to correlate with stroke severity [21]. Our 
results provide further support for this novel form of 
therapy, as they indicate that training subjects in SMR- 
modulation may have a beneficial impact on their VMIL 
skills (cf. Figure 14 (IC 9) and Figure 15). We conjecture 
that subjects with better VMIL skills require less training 
to relearn a disturbed mapping between movement goals 
and motor commands, potentially resulting in enhanced 
post-stroke motor learning. Our results further suggest 
that BCI studies on stroke rehabilitation should extend 
their focus beyond SMR-training. As we found pari- 
etal areas to be most useful for VMIL prediction, stroke 
subjects may also benefit from neurofeedback training 
that aims to enhance a //x- rhythms originating in parietal 
cortex. 
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